Getting and analyzing remotely sensed open data

2023.05.16, 2023.05.23

Our last tutorial will show you how to navigate through a STAC catalog, search within a cloud-hosted data set, and acquire some popular open satellite data sets.

Goals

1. Use Intake-STAC to browse a STAC catalog and load the data

2. TBD - Make a video showing the time lapse of the Taoyuan international airport?

Installation

We will use Intake, an interface for loading data into the scientific Python ecosystem. Here we install intake-stac, an intake-based library for loading STAC catalogs. (Intake itself will be installed along with this library because of the dependency.) In addtion to that, we will need hvplot again to visualize our data.

!pip install intake-stac hvplot

Goal 1 procedure

Let’s import the necessary modules:

import intake
import hvplot.xarray

STAC

by the STAC team, CC-BY 4.0.

SpatioTemporal Asset Catalogs (STAC) are a set of specification for describing spatialtemporal data sets, such as satellite images and spatial vector data. Satellite data sets described by the language of STAC can be more accessible because users will not spend time learning how to search and access data again and again (which is typical for the current commercial data sets). The first version of STAC was released in May 2021 and became popular within major satellite data providers. You can also check out this blogpost by the Planet Inc. for more details.

The full specification of STAC as well as numerous tutorials are available on the STAC website. Also, on the STAC intex website you can find a collection of STAC catalogs to explore.

Intake

Description about Intake. PySTAC and Intake.

The Planet disaster data

In this tutorial, we will use the Planet disaster data, a small data set that Planet Inc. prepared for showing how STAC works with the data analysis workflow. We will be looking at Hurricane Harvey, one of the most devastating natural disasters in the U.S. history. In this tutorial, we will get the data acquired by the PlanetScope satellite constellation over Houston, TX, U.S. on August 31, 2017, a few days after the hurricane brought the record-breaking rainfall.

catalog_url = 'https://www.planet.com/data/stac/catalog.json'

STAC browser

The STAC brower provides an easy way to visualize a STAC catalog.

Workflow

Now we use Intake to open the STAC cataog:

catalog = intake.open_stac_catalog(catalog_url)
catalog

planet:
  args:
    stac_obj: https://www.planet.com/data/stac/catalog.json
  description: ''
  driver: intake_stac.catalog.StacCatalog
  metadata:
    description: This catalog serves as the main root STAC Catalog for Planet Labs.
      It links to 4 small static catalogs of open data, including a small set of Planet
      Disaster Data, some CC-BY SkySat collects and the complete Spacenet 7 images
      and labels.
    id: planet
    stac_extensions:
    - https://stac-extensions.github.io/stats/v0.2.0/schema.json
    stac_version: 1.0.0
    stats:catalogs:
      count: 9
      versions:
        1.0.0: 9
    stats:collections:
      count: 7
      versions:
        1.0.0: 7
    stats:items:
      assets:
        application/geo+json: 1423
        application/json: 26
        image/jpeg: 1
        image/png: 13
        image/tiff: 2
        image/tiff; application=geotiff: 2253
        image/tiff; application=geotiff; profile=cloud-optimized: 425
        text/xml: 3
      count: 3708
      extensions:
        https://stac-extensions.github.io/eo/v1.0.0/schema.json: 396
        https://stac-extensions.github.io/label/v1.0.0/schema.json: 1423
        https://stac-extensions.github.io/projection/v1.0.0/schema.json: 385
        https://stac-extensions.github.io/raster/v1.0.0/schema.json: 3
        https://stac-extensions.github.io/raster/v1.1.0/schema.json: 371
        https://stac-extensions.github.io/view/v1.0.0/schema.json: 31
      versions:
        1.0.0: 3708
    title: Planet Root STAC Catalog
    type: Catalog

list(catalog)

['planet-stac-skysat',
 'planet-disaster-data',
 'sn7',
 'planet/fusion/14N/29E-188N',
 'education']

collection = catalog['planet-disaster-data']
collection

planet-disaster-data:
  args:
    stac_obj: https://www.planet.com/data/stac/disasters/collection.json
  description: '[Planet Disaster Data](https://www.planet.com/disasterdata/) makes
    imagery available directly to the public, volunteers, humanitarian organizations,
    and other coordinating bodies in support of the International Charter for Space
    and Major Disasters. Data is released for individual disaster events, providing
    a 30 day window pre- and post-disaster. Imagery is provided under Creative Commons
    licenses, free of charge, with either CC-BY-SA or CC-BY-NC. This STAC catalog
    provides fully public access of a very small subset of the imagery, as a proof
    of concept.'
  driver: intake_stac.catalog.StacCollection
  metadata:
    catalog_dir: ''

collection.metadata

{'type': 'Collection',
 'id': 'planet-disaster-data',
 'stac_version': '1.0.0',
 'description': '[Planet Disaster Data](https://www.planet.com/disasterdata/) makes imagery available directly to the public, volunteers, humanitarian organizations, and other coordinating bodies in support of the International Charter for Space and Major Disasters. Data is released for individual disaster events, providing a 30 day window pre- and post-disaster. Imagery is provided under Creative Commons licenses, free of charge, with either CC-BY-SA or CC-BY-NC. This STAC catalog provides fully public access of a very small subset of the imagery, as a proof of concept.',
 'stac_extensions': [],
 'title': 'Planet Disaster Data',
 'keywords': ['disaster', 'open'],
 'summaries': {'platform': ['SS02', 'SSC1', '101c'],
  'constellation': ['skysat', 'planetscope'],
  'eo:cloud_cover': {'minimum': 0, 'maximum': 24},
  'eo:gsd': {'minimum': 0.9, 'maximum': 3.7},
  'view:off_nadir': {'minimum': 0.2, 'maximum': 27.3},
  'view:sun_azimuth': {'minimum': 122, 'maximum': 231.9},
  'view:sun_elevation': {'minimum': 56.3, 'maximum': 65.1}},
 'providers': [{'name': 'Planet',
   'description': 'Contact Planet at https://www.planet.com/contact-sales/',
   'roles': ['producer', 'processor'],
   'url': 'https://www.planet.com'},
  {'name': 'Planet Disaster Team <disaster-team@planet.com>',
   'description': 'The Planet Disaster Data Team has released this data as CC-BY-SA-4.0 to help disaster response',
   'roles': ['licensor'],
   'url': 'https://www.planet.com/disasterdata/'},
  {'name': 'Google Cloud Platform',
   'description': 'Hosting is done on a GCP storage bucket owned by the Planet Disaster Data team',
   'roles': ['host'],
   'url': 'https://storage.googleapis.com/pdd-stac/'}],
 'extent': {'spatial': {'bbox': [[-96, 28, -93, 31]]},
  'temporal': {'interval': [['2017-08-28T10:00:00Z', None]]}},
 'license': 'CC-BY-SA-4.0'}

item = collection['hurricane-harvey']['hurricane-harvey-0831']['Houston-East-20170831-103f-100d-0f4f-RGB']
item

mosaic:
  args:
    chunks: {}
    urlpath: https://storage.googleapis.com/pdd-stac/disasters/hurricane-harvey/0831/Houston-East-20170831-103f-100d-0f4f-3-band.tif
  description: 3 Band RGB Mosaic
  direct_access: allow
  driver: intake_xarray.raster.RasterIOSource
  metadata:
    href: https://storage.googleapis.com/pdd-stac/disasters/hurricane-harvey/0831/Houston-East-20170831-103f-100d-0f4f-3-band.tif
    plots:
      geotiff:
        cmap: viridis
        data_aspect: 1
        dynamic: true
        frame_width: 500
        kind: image
        rasterize: true
        x: x
        y: y
    roles:
    - data
    title: 3 Band RGB Mosaic
    type: image/tiff; application=geotiff; profile=cloud-optimized
thumbnail:
  args:
    chunks: {}
    urlpath: https://storage.googleapis.com/pdd-stac/disasters/hurricane-harvey/0831/Houston-East-20170831-103f-100d-0f4f-3-band.png
  description: Thumbnail
  direct_access: allow
  driver: intake_xarray.image.ImageSource
  metadata:
    href: https://storage.googleapis.com/pdd-stac/disasters/hurricane-harvey/0831/Houston-East-20170831-103f-100d-0f4f-3-band.png
    plots:
      thumbnail:
        bands: channel
        data_aspect: 1
        flip_yaxis: true
        kind: rgb
        x: x
        xaxis: false
        y: y
        yaxis: false
    roles:
    - thumbnail
    title: Thumbnail
    type: image/png

thumbnail = item['thumbnail']
# make a note for type(thumbnail)

thumbnail.urlpath

'https://storage.googleapis.com/pdd-stac/disasters/hurricane-harvey/0831/Houston-East-20170831-103f-100d-0f4f-3-band.png'

da_thumbnail = thumbnail.to_dask()
da_thumbnail

<xarray.DataArray (y: 552, x: 549, channel: 3)>
dask.array<xarray-<this-array>, shape=(552, 549, 3), dtype=uint8, chunksize=(552, 549, 3), chunktype=numpy.ndarray>
Coordinates:
  * y        (y) int64 0 1 2 3 4 5 6 7 8 ... 543 544 545 546 547 548 549 550 551
  * x        (x) int64 0 1 2 3 4 5 6 7 8 ... 540 541 542 543 544 545 546 547 548
  * channel  (channel) int64 0 1 2

xarray.DataArray

• y: 552
• x: 549
• channel: 3

• dask.array<chunksize=(552, 549, 3), meta=np.ndarray>

  Array Chunk Bytes 887.84 kiB 887.84 kiB Shape (552, 549, 3) (552, 549, 3) Dask graph 1 chunks in 1 graph layer Data type uint8 numpy.ndarray

• Coordinates: (3)
  • y
    (y)
    int64
    0 1 2 3 4 5 ... 547 548 549 550 551

    array([  0,   1,   2, ..., 549, 550, 551])

  • x
    (x)
    int64
    0 1 2 3 4 5 ... 544 545 546 547 548

    array([  0,   1,   2, ..., 546, 547, 548])

  • channel
    (channel)
    int64
    0 1 2

    array([0, 1, 2])

• Indexes: (3)
  • y
    PandasIndex

    PandasIndex(Int64Index([  0,   1,   2,   3,   4,   5,   6,   7,   8,   9,
                ...
                542, 543, 544, 545, 546, 547, 548, 549, 550, 551],
               dtype='int64', name='y', length=552))

  • x
    PandasIndex

    PandasIndex(Int64Index([  0,   1,   2,   3,   4,   5,   6,   7,   8,   9,
                ...
                539, 540, 541, 542, 543, 544, 545, 546, 547, 548],
               dtype='int64', name='x', length=549))

  • channel
    PandasIndex

    PandasIndex(Int64Index([0, 1, 2], dtype='int64', name='channel'))

• Attributes: (0)

da_thumbnail.hvplot.image(x='x', y='y', data_aspect=1)

mosaic = item['mosaic']
da_mosaic = mosaic.to_dask()
da_mosaic

<xarray.DataArray (band: 3, y: 22094, x: 21984)>
dask.array<open_rasterio-501c37a1d96e315c20253204d9aec1ce<this-array>, shape=(3, 22094, 21984), dtype=uint8, chunksize=(3, 22094, 21984), chunktype=numpy.ndarray>
Coordinates:
  * band     (band) int64 1 2 3
  * y        (y) float64 3.524e+06 3.524e+06 3.524e+06 ... 3.447e+06 3.447e+06
  * x        (x) float64 -1.066e+07 -1.066e+07 ... -1.058e+07 -1.058e+07
Attributes: (12/13)
    transform:               (3.4638558402435815, 0.0, -10657435.586420376, 0...
    crs:                     +init=epsg:3857
    res:                     (3.4638558402435815, 3.4638558402435815)
    is_tiled:                1
    nodatavals:              (nan, nan, nan)
    scales:                  (1.0, 1.0, 1.0)
    ...                      ...
    AREA_OR_POINT:           Area
    TIFFTAG_DATETIME:        2017:09:01 15:10:49
    TIFFTAG_RESOLUTIONUNIT:  2 (pixels/inch)
    TIFFTAG_SOFTWARE:        Adobe Photoshop CC 2015.5 (Macintosh)
    TIFFTAG_XRESOLUTION:     72
    TIFFTAG_YRESOLUTION:     72

xarray.DataArray

• band: 3
• y: 22094
• x: 21984

• dask.array<chunksize=(3, 22094, 21984), meta=np.ndarray>

  Array Chunk Bytes 1.36 GiB 1.36 GiB Shape (3, 22094, 21984) (3, 22094, 21984) Dask graph 1 chunks in 2 graph layers Data type uint8 numpy.ndarray

• Coordinates: (3)
  • band
    (band)
    int64
    1 2 3

    array([1, 2, 3])

  • y
    (y)
    float64
    3.524e+06 3.524e+06 ... 3.447e+06

    array([3523817.157214, 3523813.693358, 3523810.229502, ..., 3447297.117847,
           3447293.653991, 3447290.190136])

  • x
    (x)
    float64
    -1.066e+07 ... -1.058e+07

    array([-10657433.854492, -10657430.390637, -10657426.926781, ...,
           -10581294.839268, -10581291.375412, -10581287.911556])

• Indexes: (3)
  • band
    PandasIndex

    PandasIndex(Int64Index([1, 2, 3], dtype='int64', name='band'))

  • y
    PandasIndex

    PandasIndex(Float64Index([ 3523817.157214148, 3523813.6933583077, 3523810.2295024674,
                   3523806.765646627,  3523803.301790787, 3523799.8379349466,
                  3523796.3740791064,  3523792.910223266,  3523789.446367426,
                  3523785.9825115856,
                  ...
                  3447321.3648382085, 3447317.9009823683, 3447314.4371265285,
                   3447310.973270688,  3447307.509414848, 3447304.0455590077,
                  3447300.5817031674,  3447297.117847327,  3447293.653991487,
                  3447290.1901356466],
                 dtype='float64', name='y', length=22094))

  • x
    PandasIndex

    PandasIndex(Float64Index([-10657433.854492456, -10657430.390636615, -10657426.926780775,
                  -10657423.462924935, -10657419.999069095, -10657416.535213254,
                  -10657413.071357414, -10657409.607501574, -10657406.143645734,
                  -10657402.679789893,
                  ...
                  -10581319.086258944, -10581315.622403104, -10581312.158547264,
                  -10581308.694691423, -10581305.230835583, -10581301.766979743,
                  -10581298.303123903, -10581294.839268062, -10581291.375412222,
                  -10581287.911556382],
                 dtype='float64', name='x', length=21984))

• Attributes: (13)

  transform :

  (3.4638558402435815, 0.0, -10657435.586420376, 0.0, -3.4638558402435815, 3523818.889142068)

  crs :

  +init=epsg:3857

  res :

  (3.4638558402435815, 3.4638558402435815)

  is_tiled :

  1

  nodatavals :

  (nan, nan, nan)

  scales :

  (1.0, 1.0, 1.0)

  offsets :

  (0.0, 0.0, 0.0)

  AREA_OR_POINT :

  Area

  TIFFTAG_DATETIME :

  2017:09:01 15:10:49

  TIFFTAG_RESOLUTIONUNIT :

  2 (pixels/inch)

  TIFFTAG_SOFTWARE :

  Adobe Photoshop CC 2015.5 (Macintosh)

  TIFFTAG_XRESOLUTION :

  72

  TIFFTAG_YRESOLUTION :

  72

da_mosaic_subset = da_mosaic.sel(x=slice(-10641000, -10638000), y=slice(3474000, 3471000))
da_mosaic_subset.hvplot.rgb(x='x', y='y', data_aspect=1)